WavSpA: Wavelet Space Attention for Boosting Transformers' Long Sequence Learning Ability

Transformer and its variants are fundamental neural architectures in deep learning. Recent works show that learning attention in the Fourier space can improve the long sequence learning capability of Transformers. We argue that wavelet transform shall be a better choice because it captures both position and frequency information with linear time complexity. Therefore, in this paper, we systematically study the synergy between wavelet transform and Transformers. We propose Wavelet Space Attention (WavSpA) that facilitates attention learning in a learnable wavelet coefficient space which replaces the attention in Transformers by (1) applying forward wavelet transform to project the input sequences to multi-resolution bases, (2) conducting attention learning in the wavelet coefficient space, and (3) reconstructing the representation in input space via backward wavelet transform. Extensive experiments on the Long Range Arena demonstrate that learning attention in the wavelet space using either fixed or adaptive wavelets can consistently improve Transformer's performance and also significantly outperform learning in Fourier space. We further show our method can enhance Transformer's reasoning extrapolation capability over distance on the LEGO chain-of-reasoning task

Rate-Splitting with Hybrid Messages: DoF Analysis of the Two-User MIMO Broadcast Channel with Imperfect CSIT

Most of the existing research on degrees-of-freedom (DoF) with imperfect channel state information at the transmitter (CSIT) assume the messages are private, which may not reflect reality as the two receivers can request the same content. To overcome this limitation, we consider hybrid private and common messages. We characterize the optimal DoF region for the two-user multiple-input multiple-output (MIMO) broadcast channel with hybrid messages and imperfect CSIT. We establish a three-step procedure for the DoF converse to exploit the utmost possible relaxation. For the DoF achievability, since the DoF region has a specific three-dimensional structure w.r.t. antenna configurations and CSIT qualities, by dividing CSIT qualities into cases, we check the existence of corner point solutions, and then design a hybrid messages-aware rate-splitting scheme to achieve them. Besides, we show that to achieve the strictly positive corner points, it is unnecessary to split the private messages into unicast and multicast parts because the allocated power for the multicast part should be zero. This implies that adding a common message can mitigate the rate-splitting complexity of private messages.Comment: 32page

RecycleGPT: An Autoregressive Language Model with Recyclable Module

Existing large language models have to run K times to generate a sequence of K tokens. In this paper, we present RecycleGPT, a generative language model with fast decoding speed by recycling pre-generated model states without running the whole model in multiple steps. Our approach relies on the observation that adjacent tokens in a sequence usually have strong correlations and the next token in a sequence can be reasonably guessed or inferred based on the preceding ones. Experiments and analysis demonstrate the effectiveness of our approach in lowering inference latency, achieving up to 1.4x speedup while preserving high performance.Comment: Technical Repor

Repair of Adult Mammalian Heart After Damages by Oral Intake of Gu Ben Pei Yuan San

Adult mammalian heart repair after myocardial damage is highly inefficient due to the post-mitotic nature of cardiomyocytes. Interestingly, in traditional Chinese medicine (TCM), there are reported effective treatments of myocardial infarction (MI) and heart failure in adult humans by oral intake of a TCM concoction named Gu Ben Pei Yuan San (GBPYS), which is composed of Panax ginseng, velvet antler, Gekko gecko Linnaeus tail, human placenta, Trogopterus dung, Panax notoginseng, and amber. We fed mice with GBPYS after myocardial damages through everyday self-feeding. We then examined the effect of everyday oral intake of GBPYS on improving cardiac function and myocardial repair in adult mice after apical resection or MI. We found that long-term oral intake of GBPYS significantly improved cardiac function after myocardial damages in adult mice. BrdU, phospho-histone 3, and AuroraB staining indicated increased cell proliferation at the border zone of MI after TCM feeding. GBPYS feeding reduced organ inflammation, induced angiogenesis, and is non-toxic to mice after long-term oral intake. Further, serum derived from TCM-fed MI rats promoted division of both neonatal rat cardiomyocytes and human induced pluripotent stem cell (iPSC)-derived cardiomyocytes in vitro. Oral intake of GBPYS improved heart repair after myocardial damages in adult mice. Our results suggest that there are substances present in GBPYS that help improve adult mammalian heart repair after MI. Also, it could be a good choice of non-invasive alternative therapy for myocardial damages and heart failure after rigorous clinical study in the future

The Generalized Degrees-of-Freedom of the Asymmetric Interference Channel with Delayed CSIT

In this paper, we investigate the generalized degrees-of-freedom (GDoF) of the asymmetric interference channel with delayed channel state information at the transmitter (CSIT), where each transmitter has two antennas, each receiver has one antenna, and the strength for each interfering link can vary. The optimal sum-GDoF is characterized by matched converse and achievability proof. Through our results, we also reveal that in our antenna setting, the symmetric GDoF lower bound in [Mohanty et. al, TIT 2019] can be elevated, and the symmetric GDoF upper bound in [Mohanty et. al, TIT 2019] is tight in fact.Comment: Accepted by IEEE ISIT 202

TriNymAuth: Triple Pseudonym Authentication Scheme for VANETs Based on Cuckoo Filter and Paillier Homomorphic Encryption

In VANETs, owing to the openness of wireless communication, it is necessary to change pseudonyms frequently to realize the unlinkability of vehicle identity. Moreover, identity authentication is needed, which is usually completed by digital certificates or a trusted third party. The storage and the communication overhead are high. This paper proposes a triple pseudonym authentication scheme for VANETs based on the Cuckoo Filter and Paillier homomorphic encryption (called TriNymAuth). TriNymAuth applies Paillier homomorphic encryption, a Cuckoo Filter combining filter-level and bucket-level, and a triple pseudonym (homomorphic pseudonym, local pseudonym, and virtual pseudonym) authentication to the vehicle identity authentication scheme. It reduces the dependence on a trusted third party and ensures the privacy and security of vehicle identity while improving authentication efficiency. Experimental results show that the insert overhead of the Cuckoo Filter is about 10 μs, and the query overhead reaches the ns level. Furthermore, TriNymAuth has significant cost advantages, with an OBU enrollment cost of only 0.884 ms. When the data rate in VANETs dr≤ 180 kbps, TriNymAuth has the smallest total transmission delay cost and is suitable for shopping malls and other places with dense traffic

Reconfigurable Intelligent Surface Assisted OFDM Relaying: Subcarrier Matching with Balanced SNR

Reconfigurable intelligent surface (RIS) is a promising solution to enhance the performance of wireless communications via reconfiguring the wireless propagation environment. In this paper, we investigate the joint design of RIS passive beamforming and subcarrier matching in RIS-assisted orthogonal frequency division multiplexing (OFDM) dual-hop relaying systems under two cases, depending on the presence of the RIS reflected link from the source to the destination in the first hop. Accordingly, we formulate a mixed-integer nonlinear programming (MINIP) problem to maximize the sum achievable rate over all subcarriers by jointly optimizing the RIS passive beamforming and subcarrier matching. To solve this challenging problem, we first develop a branch-and-bound (BnB)-based alternating optimization algorithm to obtain a near-optimal solution by alternatively optimizing the subcarrier matching by the BnB method and the RIS passive beamforming by using semidefinite relaxation techniques. Then, a low-complexity difference-of-convex penalty-based algorithm is proposed to reduce the computation complexity in the BnB method. To further reduce the computational complexity, we utilize the learning-to-optimize approach to learn the joint design obtained from optimization techniques, which is more amenable to practical implementations. Lastly, computer simulations are presented to evaluate the performance of the proposed algorithms in the two cases. Simulation results demonstrate that the RIS-assisted OFDM relaying system achieves sustainable achievable rate gain as compared to that without RIS, and that with random passive beamforming, since RIS passive beamforming can be leveraged to recast the subcarrier matching among different subcarriers and balance the signal-to-noise ratio within each subcarrier pair.Comment: Submitted to IEE